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Abstract 

Introduction. The prediction module generates possible future trajectories of dynamic objects that enables a self-driving 
vehicle to move safely on public roads. However, all modern prediction methods evaluate their performance only under 
urban conditions and do not consider their applicability to the domain of rural roads. This work examined the adaptability 
of existing methods to work under rural unstructured conditions and suggested a new, improved approach. 

Materials and Methods. As a solution, we propose to use a neural network that includes the following submodules: a 
graph-based scene encoder, a multimodal trajectory decoder, and a trajectory filtering module. Another proposed feature 
is to use an adapted loss function that penalizes the network for generating trajectories that go beyond the drivable area. 
These elements use standard practices for solving the prediction problem and adapting it to the domain of rural roads. 
Results. The presented analysis described the basic features of the prediction module in the rural road domain, showed a 
comparison of popular models, and discussed its applicability to new conditions. The paper describes the new approach 
that is more adaptive to the considered domain of study. A simulation of the new domain was performed by modifying 
existing public datasets. 

Discussion and Conclusion. Comparison to other popular methods has shown that the proposed approach provides more 
accurate results. The disadvantages of the proposed approach were also identified and possible solutions were described. 
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AHHOTalna 

Beedenue. buaroyapa MoxyI10 MpornosMpoBaHuA TpaeKTOpHii ABWKeHHA THHAaMHYeCKUX OOBeKTOB OeciHJOTHBIM 
aBTOMOOMJIb CllocobeH Oe3s0MacHo ABHraTbedA 10 WoporaM oOulero Mob30BaHHA. OWHAKO BCe COBPeMeHHbIe MeTObI 
TIpOrHO3HpOBaHHA OLWCHHBAaIOT MPOW3BOUTeIbHOCTh TOJIbKO B TOPOCKHX YCJOBHAX MH He PaCCMaTPHBalOT CBOIO 
IIPHMe€HUMOCTb K JOMeHy Mpoceso4HBIx AZopor. Len DZaHHoro UcceqOBaHHA 3aKkOUAeTCA B AHaIM3e alalTHBHOCTH 
CYUIeCTBYIOWJMX MeTONOB MIpOrHo3HpoBaHHA MU pa3paOoTKe MOAXoya, KOTOpbIM OyeT AeEMOHCTpHpoBaTh JIy4LLy10 
IIPOH3BOHTEJILHOCT IIpH paOoTe B HOBBIX YCJIOBHAX. 

Mamepuanoi u memoooi. B kauectBe pelieHua MpeslaraeTcA MCMONb30BaTb HEHPOHHYIO CeTb, BKJIOYAIOLIyIO B cebs 
crelyroulHe MOAMOAyIM: rpapoBblit KOAMPOBLIHK CI[CHbI, MYJIbTHMOMaIbHbIM JeKOAMpOBWIHK TpaeKTOPpHi, MOJyJIb 
(puIbTpalyun Tpaektopui. TawKe mpesiaraeTca NPHMeHUTb alanTHupoBaHHylo PYHKUMIO MOTepb, KOTOpad WITpadpyer 
CeTb 3a TeHepallMiO TpacKTOpH, BEIXOAAMUX 3a TpaHHUbl AOpoxkHoro MowoTHa. JJaHHble IIeMeHTbI 3aeHCTByIOT 
paciipocTpaHéHHble MpaKTHKH PeWIeHHA 3aa4n NPOTrHO3MPOBAHHA, a TakOKe aflalITHpyIOT eé WI WOMeHa MIpOcesO YHBIX 
Wopor. 

Pezynsmamot uccredosanua. ([poananu3upoBaHbl OCHOBHBIe OTIIMYHA H YCIOBHA paOOThI MOJYIA IpOrHOSHpOBaHHA B 
yCJIOBHAX MIPpOCeIOUHBIX Jopor. BenosHeHa CHMYJIAUHA HOBOTO JOMeHa IlyTEM MOAMUuKAalHH CyLecTBYIOWIMX HabopoB 
aHHbix. IpopexeHo cpaBHeHHe MOMMyIAPHbIX MeTOJOB MporHo3sHpoBaHHA HM OWeHeHa HX IPHMCHHMOCTb K HOBbIM 
ycnoBusaM. IIpezcTaBiieH HOBbIM, OosIee alanTHBHbIM K HOBOMY JOMeHYy, MOAXod. 

O6cyscdenue u 3akimio4uenue. IIpopeqeHHoe cpaBHeHve C JIpyrHMM MOMyJIApHbIMH MeTOJAaMM TOKa3bIBaeT, 4TO 
Tipesox*KeHHOe aBTOpaMH pelieHHe obecreauBaeT Oosee TOUHbIC pe3yIbTaTbI MporHo3sMpoBaHHna. TawKe Obi 


BBIABJICHbI HELOCTATKH MpeAIO#xKCHHOTO MOAXOa HW OMMCaHbI BO3MO2KHBIC IlyTH HX YCTpaHeHHsA. 


KoroueBbie cJI0Ba: TIpOrHOSHpOBaHHe TpaeKTopui, TIPOrHO3HpOBaHHe NOBeTCHHA, HelpoHHble ceTH, OeciHHIOTHBIC 


aBTOMOOMJIN, MCKYCCTBeHHbIi UHTCJVICKT, ABTOHOMHBIC aBTOMOOMIM 


BuaarogapHocrn: aBTOPbI BbIpaxKaroT OlaroyapHOCTb WeHTpy OecHHJIOTHBIX TEXHOJIOMM yHHBepcuteta VWnHonomne 3a 


TIOMOIMb B HpOBeTeCHHH HCCIeTOBaHHA. 


Aaa untTupopanna. Veaxos C.A., Pamuy b. IIporno3supopanue noBpeyeHua y4acTHHKOB JOPOXKHOTO BYWKeHHA B 
YCIOBHAX IpOCeNOYHBIX WOpor Ayia OeciMMOTHBIX aBTOMOOUIeH. Advanced Engineering Research (Rostov-on-Don). 
2023;23(2):169-179. https://doi.org/10.23947/2687-1653-2023-23-2-169-179 


Introduction. The latest achievements in the field of artificial intelligence (AJ) are being actively implemented in 
various areas of activity. One of such achievements is autonomous vehicles (AV). The current research was aimed at 
creating algorithms that allow AV to move safely on public roads. This will significantly reduce the number of road 
accidents [1]. 

The scientific community has already identified the basic modules of an autonomous vehicle. One of them is a 
system for predicting the future behavior of road users (agents) [2]. A clear understanding of how the environment will 
develop and in which direction dynamic objects (pedestrians, cars, cyclists) will move is urgently needed for AV to 
search and use a safe and effective trajectory of movement. 

Numerous scientific papers are devoted to the problem of predicting such trajectories [3-12]. However, there is 
currently no active research on the application of existing methods outside of urban conditions. And this is extremely 


important, since autonomous cars will be used on country roads, too [13]. Urban conditions are highly structured: cars 
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mostly follow traffic lanes, and pedestrians move through special zones. In this sense, the area of country roads is the 
complete opposite, which means that it will have additional difficulties during development. In the given paper, attention 
is focused on these difficulties: the existing predicting methods and their applicability to new, less structured conditions 
are considered. 

The objective of the study involved: 

—analysis of the major differences and working conditions of the prediction module under the conditions of country roads; 

— simulation of a less structured country road domain by modifying the existing datasets; 

— comparison of modern prediction methods, including their applicability to new conditions; 

— description of the new approach and proof of its higher accuracy in comparison to other prediction methods. 

Materials and Methods. At first glance, it would seem that the domain presented is a simpler version of urban 
conditions due to the fact that country roads are characterized by less traffic. However, the absence of complex multi- 
level junctions, special traffic-free zones, a large number of signs, markings, etc., makes the domain of country roads less 
structured, i.e., fewer rules and specific traffic patterns increases the randomness and reduces the predictability of the 
behavior of cars and pedestrians. 

The following features of the country road domain influence strongly on the selection of the architecture of the 
prediction module: 

— crossroads. Undoubtedly, they are more simple on country roads in comparison to urban ones, but at the same time 
this fact of simplicity means that the model must take into account multimodality and assess the probability of choosing 
each possible direction of movement at the crossroad when the agent approaches it; 

— country roads do not have lane markings, pedestrian crossings, bike paths, etc. Instead, the HD map will contain 
only information about the boundaries of the roadway. Therefore, the stage of encoding the scene should take into account 
this feature to describe the surrounding context more effectively; 

— pedestrians and cyclists will move along the same road with conventional and autonomous vehicles. Therefore, the 
model should be adaptive for predicting the future trajectories of both cars and pedestrians/cyclists. 

The prediction module implies the presence of AV recognition, tracking and localization systems and their accurate 
operation. The authors of the article use the Argoverse dataset, which stores the required records of the operation of all 
systems in a convenient form [14]. 

The dataset consists of recordings of road scenes observed on the streets of Miami and Pittsburgh, USA. Each of the 
entries contains a local part of the terrain map (lane boundaries, roads, pedestrian crossings) and a list of all recognized 
agents, including the current position and movement history of each of them. Each of the records is divided into two parts: 
two seconds of the observation history and three subsequent seconds for which prediction is made (prediction horizon). 
Data on the future movement of objects is also available and used to calculate the accuracy of prediction methods and 
model training. 

Information about agents is presented in a discrete format. The time interval between measurements is fixed, in this 
work it is equal to 0.1 seconds (10 Hz). 

For each moment of time ¢, the module receives the observation history or for each detected agent i. The observation 
history consists of the agent's current and past states, where each of the states s, is a 2D position in the global coordinate 
system. The authors make the assumption that the height information is redundant. 

The dataset also provides access to an HD map that contains information about road borders and roadway, pedestrian 
crossings. To simulate the domain of country roads, the dataset was modified in such a way as to exclude all information 
from road maps, except for the boundaries of the roadway D. This reduces the amount of information about the road 
context and complicates the task of prediction. 


Hence, the context of the scene is represented as 
CHATS? BP SPD), (1) 


where k — the total number of tracked agents on the scene. 
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This approach implies predicting the trajectory for only one agent per execution, therefore further S,’ is treated as 


S” for simplification. To generalize the model for all recognized agents, it is required to repeat the proposed approach 
for all k agents on the scene. The agent for which the prediction is currently being made is considered a target agent. 


To assess the accuracy of prediction methods, the dataset contains recorded future trajectories S/ for each target agent: 
SS i Sina aiel (2) 


where H indicates the number of next time steps. In this case, parameter H will be equal to 30, since the planning horizon 


is three seconds with a sampling frequency of 10 Hz. 

The domain of the prediction module is multimodal, i.e., the future behavior of agents may differ significantly in 
absolutely identical traffic situations. Let us say, a car approaching a crossroad may continue straight ahead or make a 
turn. To take this into account, it is required to generate M possible future trajectories and M probabilities of each of them 
at the model output. 


Therefore, the purpose of the prediction module is to create function f, that takes the context of the scene c as input 


and generates M pairs of possible future trajectories and their probabilities: 
FC) = SF Si o9St PaDae sat (3) 


Here, at least one generated trajectory S/ should be as close as possible to the real trajectory S/ , and the probability 
of its execution p should be close to unity. 
Model architecture. The proposed approach involves the use of a neural network consisting of submodules of scene 


encoding, decoding and filtering trajectories. The architecture of the system is shown in Figure 1. 
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Fig. 1. System architecture 


A neural network adapted to the new conditions, based on a vector representation, is responsible for encoding the 
scene. This selection is due to the fact that on country roads, the HD map will contain a limited amount of information 
(only the roadway boundaries and the history of observations of dynamic objects). Popular methods represent the context 
of a road scene c in an image format and process it using convolutional neural networks. However, vector coding avoids 
the overhead associated with image generation [4—5]. 

The presented encoder is based on the VectorNet model, but its input data format has been modified to receive 
information only about the boundaries of the roadway and the state of the agents. [3]. This encoder represents the 
boundaries of the road and the state of agents using polylines, which are further processed by a graph neural network. 
This provides encoding the interaction between polylines. Details of the implementation are described in paper [3]. 

A trajectory decoder is a task of regressing several possible trajectories and generating a set of probabilities. To solve 
this problem, a multilayer perceptron model is used. The decoder implementation is inspired by the MTP model [4], 
however, the authors of the article propose a different formula for calculating the best trajectory m* from the set of 
M trajectories. It is also proposed to use an additional mechanism that penalizes the model for predictions that go beyond 


the area of movement. 
The authors of the original MTP model propose to train a multilayer perceptron using the loss function that represents 


the sum £,,, and £,,,,,, where: 


lass ? 


b= ES" 85) () 


Teg 
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M 
Ltass = -yi log Pn (5) 
m=1 


In this case, £,,, —- mean-square error between the real trajectory S ! and the best trajectory m* of M generated. 


e(s'.s/)=2>(s-s,) » ©) 


i=l 
where s;— the agent's actual future position at time i, and 5 — the predicted future state of the best trajectory m’*. 


L, 


asx ~~ 1088 function based on cross entropy, which increases the probability of executing the best of the predicted 
trajectories m* to 1 and reduces the probability of other trajectories to 0. 

[, is a binary indicator equal to 1, if condition c is true, and 0 — otherwise. 

In the original article, the best of the predicted trajectories m* is defined as the one that has the minimum value of the 


root-mean-square error in comparison to the real trajectory: 


m=argmin (5? 82). (7) 


m 
m 


The authors of the article suggest using the following modification: 


m 


m=argmin (sts!) (8) 


meA 


where A — subset of generated trajectories that has a similar final direction to the real trajectory S/ . 

The idea is to remove from consideration trajectories in which the final direction of the agent differs significantly 
from the direction in the real trajectory when calculating the best trajectory m*. If the difference in directions features 
less than certain threshold y, then the generated trajectory is considered correct, i.e., mA. In the case under consideration 
y=30°. Therefore, the best trajectory m* should have a similar final direction and the lowest value of the loss function. 

This work also involves prior knowledge of the domain to achieve greater convergence of the model [15]. Since only 
information about the roadway boundaries is available from the HD map when driving in the domain of country roads, 


an additional variable is introduced — £,, into the loss function. Thanks to it, the model will penalize the predicted 


trajectories that go beyond the road in cases where at least one state is s; € D. The model penalizes only the best trajectory, 


since only in this case, it is possible to determine the direction of error reduction by approximating the best of the generated 
trajectories m* to the real trajectory S’. 


Thus, £,, is defined as: 


12 + \? 
Lin == YI.-(5-51) ’ (9) 
where /, is equal to 1, if s, ¢ D , and 0 — otherwise. 


The final loss function is defined as 


L= Lig tO Lo +B Lips (10) 


lass reg 
where a and B — neural network hyperparameters used for training. In this case, both of these parameters are equal to 0.5. 

To filter similar and duplicate trajectories, the proposed approach uses the filtering of a finite set of trajectories M at 
the final stage. This module is required because in some cases, the number of possible agent trajectories may be less 
than M, e.g., when a car is moving along a straight road at a constant speed, the model can generate only one trajectory: 
the car continues to move straight. However, the need to generate exactly M trajectories will result in the situation when 
all predictions are similar to each other. 

The proposed filtering is based on the final direction and positions of states s;: if the direction and the sum of the 
deviations between states s; of the real and generated trajectories are less than the threshold value o, then the trajectories 
are considered similar. The authors average each state of the trajectories and sum up the probabilities of the trajectories p;. 

This approach was implemented in the Python programming language on the PyTorch deep learning framework. The 
model was trained on GeForce RTX 2080 Ti graphics card for 40 epochs, the training took four hours. 
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Research Results. To assess the accuracy of prediction models, this section applies widely used metrics for the 
trajectory prediction problem: average displacement error, ADE, final displacement error, FDE [6], MissRate (MR), and 
Offroad rate (OR). 

For multimodal cases with the generation of several trajectories, ADE and FDE are taken as the minimum ADE and 
FDE among M trajectories (the trajectory with the lowest metric value) [5]. 

The prediction is considered “missed” if ADE metric of the generated trajectory is more than two meters. OR metric 


is calculated as the percentage of trajectories in which at least one state s; goes beyond the range of motion D. 
To visualize the context of scene c, as well as the real and predicted future trajectories Sand S’ a script in the Python 


programming language was implemented using the Matplotlib library. 

This section compared the operation of several different methods in the case of an unstructured domain. The following 
methods were used in comparison: 

— Kalman filter; 

— Single trajectory output — the proposed scene encoder with the generation of a single trajectory; 

— Fixed set classification — the proposed scene encoder with the reduction of the task to classification among 
predefined trajectories: by sets of 64 and 415 predefined trajectories; 

— Proposed approach. 

Table 1 presents comparison of the accuracy of the methods when working under unstructured conditions. Several 


methods are compared, including the proposed approach. 


Table 1 
Comparison of models in the unstructured domain of work 

Method Modes | ADE, FDE, ADEs FDE, MR2, MR2. OR 
Kalman filter 1 3.78 8.05 3.78 8.05 0.89 0.89 5.89 
Single trajectory output 1 3.12 6.75 3.12 6.75 0.89 0.89 3.26 
Fixed set classification 415 3.27 7.00 1.74 3:57 0.84 0.52 3.61 
Fixed set classification 64 2.6 5.63 1.52 2.91 0.82 0.49 2.58 
Proposed approach 6 2.36 5.29 1.32 2.55 0.78 0.38 1.84 


Kalman filter. The simplest way to predict behavior is to obtain the current state of the object (current lane, speed, 
direction, etc.) and extend this state to future steps based on some assumptions, e.g., that the car will continue to follow 
its lane or will have a constant speed and/or acceleration. Another popular method for such tasks is to use the Kalman filter [12]. 

According to Table 1, the Kalman filter works worse than all the presented methods based on neural networks. 

Figure 2 shows two cases. In the first case, the Kalman filter successfully performs prediction because the vehicle is 
moving straight, without any turns or speed variation. In the second case, the Kalman filter mispredicts due to lack of 


knowledge about the context of the traffic situation. 
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Fig. 2. Example of predictions using the Kalman filter. Dotted lines — roadway boundaries, 


red lines — target agent with history of observations, blue — other agents, green — real trajectory, yellow — predicted 


trajectory, red crosses indicate predicted states outside the roadway 
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Single trajectory output. This method involves the use of a graph scene encoder, which is identical to the one used in 
the proposed approach. The output of the network implies the generation of only one trajectory. This model is trained 
using the root-mean-square loss function. 

As shown in Table 1, the neural network, even with the generation of a single trajectory, demonstrates better results 
in comparison to the Kalman filter. 

Figure 3 shows the visualization of this prediction method operation. The image on the left shows that the model can 
successfully predict the agent's turn. The image on the right shows that generating one trajectory is not enough. The neural 
network tries to imagine both possible outcomes: going straight and turning right. As a result, the model outputs the 


average of the two outcomes. 
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Fig. 3. Example of generating a single trajectory. Red line shows the target agent with 


history of observations, green — real trajectory of movement, yellow — predicted trajectory 


Fixed set classification. The implementation was inspired by the CoverNet prediction method [5]. This model consists 
of a proposed vector scene encoder, followed by a different trajectory decoder. The decoder is a classification task based 
on a predefined set of trajectories consisting of physically realizable vehicle trajectories with sufficient coverage. Two 
sets were created for experiments: of 415 and 64 possible trajectories. The second set has the same coverage as the first, 
but provides a lower density of trajectories. Detailed information about the sets of trajectories is contained in paper [5]. 

The visualization of the work is shown in Figure 4. The classification model successfully copes with multimodality at 


crossroads, but in some cases, the lack of sufficient coverage by a set of trajectories negatively affects the results. 
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Fig. 4. Example of prediction using a classification model. Red lines represent the target agent < 

with history of observations, green lines — real trajectory. M predicted trajectories a 

with different probability of execution pi are presented using red-yellow hues x 

| 

As shown in Table 1, this method works more accurately than generating a single trajectory, but worse than the Be 
proposed approach. In addition, increasing the density of the set of trajectories by using a set of 415 trajectories did not E 
improve the results. The authors attribute this to the presence of noise in the dataset, which comes from the tracking = 


system used in the data collection. 
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Proposed approach. The proposed approach eliminates the disadvantages of all the methods described above. This is 
a multimodal forecasting method that does not suffer from the limitations of a predefined set of trajectories. 
Moreover, according to Table 1, the proposed approach surpasses all other methods in all indicators. As shown in 


Figure 5, the method successfully captures two possible outcomes at the crossroad: driving straight or making a turn. 
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Fig. 5. Example of prediction using a classification model. Red lines represent the target agent 
with history of observations, green lines — real trajectory. M predicted trajectories 


with different probability of execution p; are presented using red-yellow hues 


Figure 6 shows an example of filtering similar trajectories in the case of a single possible outcome. The probability 
that the agent will complete the initiated turn is close to 1, since he is already in the process of turning. Therefore, in this 


case, the probability of other outcomes is close to 0. The proposed module successfully filters similar trajectories. 
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Fig. 6. Filtering effect. The entire set of predictions is shown on the left, and only filtered set — on the right 


Limitations. Although the authors of the original article on the MTP model [4] indicate that their method solves 
the problem of mode collapse, the experiments conducted by the authors of this article do not confirm this. The 
problem still occurs in some cases. It is assumed to be due to the following features: the loss function does not 
penalize the neural network for generating all possible trajectories that the target agent can execute, as long as the 
best of them is as close as possible to the real trajectory. But also, the model does not encourage the network in any 
way to predict a variety of possible trajectories. Therefore, it is advantageous for the network to make several similar 
predictions in one direction, in which it is more confident than to make one prediction for each possible trajectory. 

One of the possible solutions to this problem may be the use of a trajectory decoder presented in the TnT, 
DenseTnT models [10—11], which imply the generation of final goals at the first stages of work. In these models, 
all possible final goals for the agent are generated first, and then trajectories that describe the movement from the 
starting position to each of the goals, are generated. This provides filtering out similar final goals in the early stages, 


and preventing the mode collapse. 
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Discussion and Conclusion. In the work performed, modern methods of solving the trajectory prediction 
problem are investigated. The adaptability of the methods to unstructured road conditions — country roads, is 
considered. Insufficient accuracy of the methods is established, and a new approach to predicting is proposed. 

The proposed approach is based on the VectorNet and MTP models, but has been adapted for the country road 
domain. In addition, a trajectory filtering module and an additional mechanism for the loss function, which penalizes 
trajectories for going beyond the movement zone, are proposed. 

The presented comparison shows that the proposed approach is superior to other popular methods. 

Limitations of the MTP approach have been identified: the output data still tends to mode collapse. The 
suggestion for further modifications is to use methods that generate the final goal at the early stages of prediction 


and thus are less susceptible to regime collapse. 
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3aAGNEHHbIU 6KIAO CoaBmMopos: 


C.A. UspaHos — cbopMupoBaHue OCHOBHOM KOHUCHUMN, Wes UW 3aaduW UCCCTOBAHHA, IpOBeeHHe pacueTosB, 


TLOArOTOBKa TeKCTa, dopMupoBaHHe BbIBOJIOB. 


Bb. Pammuy — Hay4Hoe pyKOBOJCTBO, aHaJIH3 pe3yJIbTaTOB YccleqOBaHuh, BbITBUXKCHHE BO3SMOXKHDHIX FHIIOTC3, 


opaOorTka TekctTa. 


Konugdauxm uHMepecos6. ABTOPHI 3aABJIAIOT 06 OTCYTCTBHH KOHJIMKTa MHTCpecos. 


Bce aemopoi npowumaau u odobpusu OKOHYAMeAbHbIU BapUaHM pyKonuUcU. 
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